Spatial Cognition and Computation The Arrow-Semantics Interpreter

People often sketch diagrams to facilitate their communication. If computers understood such diagrams, people could interact with information systems more intuitively using similar multi-modal dialogs that include sketched diagrams. It remains a challenging problem, however, to make computers interpret arrow symbols, which are a frequent and versatile ingredient of such diagrams. This paper develops an algorithm for deducing possible semantic roles of arrow symbols. The Arrow-Semantics Interpreter (ASI) analyzes the elements that surround an arrow symbol and their spatial arrangements. It distinguishes four different semantic roles of arrow symbols—orientation, behavioral description, annotation, and association—each with distinctive requirements on the arrangement of the surrounding elements. The combination of these properties, together with rules for optional elements, yields the potential semantic roles of each arrow symbol from the pattern of its surrounding elements. The assessment shows the ASI derives for each arrow symbol only 1.31 potential semantic roles, which include for 79% of the analyzed diagrams the correct answer. This result indicates that the patterns of elements around arrow symbols are highly useful for deriving the interpretation of arrow symbols.